Device and method with image matching

ABSTRACT

An image matching method includes extracting, from a first image of an object, a landmark patch including a landmark point of the object; extracting, from a second image of the object, a target patch corresponding to the landmark patch; and determining a target point in the second image corresponding to the landmark point based on a matching between the landmark patch and the target patch.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of Application No. 16/148,129 filedon Oct. 1, 2018, which claims the benefit under 35 USC § 119(a) ofKorean Patent Application No. 10-2018-0017303 filed on Feb. 12, 2018 inthe Korean Intellectual Property Office, the entire disclosure of whichis incorporated herein by reference for all purposes.

BACKGROUND 1. Field

The following description relates to an image matching technology.

2. Description of Related Art

Recently, the importance of security authentication has been increasingas a result of the ongoing development of various mobile devices andwearable devices, such as smartphones. An authentication technologyusing biometrics authenticates a user using a fingerprint, an iris, avoice, a face, a blood vessel, etc. Biological properties used forauthentication are unique to a person, are convenient to carry, andremain stable for the life of the person. Also, biological propertiesare difficult to appropriate or counterfeit.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In one general aspect, there is provided an image matching methodincluding: extracting, from a first image of an object, a landmark patchincluding a landmark point of the object; extracting, from a secondimage of the object, a target patch corresponding to the landmark patch;and determining a target point in the second image corresponding to thelandmark point based on a matching between the landmark patch and thetarget patch.

The method may further include acquiring, using a color image sensor, acolor image as the first image; and acquiring, using an infrared (IR)image sensor, an IR image as the second image.

The method may further include determining, based on the target point,whether the object is an anatomical structure of a user.

The method may further include allowing access, through a user interfaceof a device, to one or more features of the device in response to theobject being determined to be a live anatomical structure and/or arecognized user.

The extracting of the target patch may include determining the targetpatch in the second image based on a location of the landmark patch inthe first image.

The extracting of the target patch may include extracting the targetpatch in response to the landmark point being detected in a determinedregion of the first image.

The extracting of the target patch may include determining thedetermined region based on a difference between a field of view (FOV) ofa first image sensor used to capture the first image and an FOV of asecond image sensor used to capture the second image.

The determining of the target point may include retrieving, from thetarget patch, a partial region that matches the landmark patch; anddetermining a center point of the retrieved partial region as the targetpoint.

The retrieving of the partial region may include calculating asimilarity level between the landmark patch and each of a plurality ofpartial regions of the target patch; and determining a partial regionwith a highest calculated similarity level among the plurality ofpartial regions as the partial region that matches the landmark patch.

The calculating of the similarity level may include calculating acorrelation level between values of pixels included in each of theplurality of partial regions of the target patch and pixels included inthe landmark patch as the similarity level.

The first image may be a color image and the second image may be aninfrared (IR) image; and the extracting of the landmark patch comprises:selecting a channel image from the first image; and extracting thelandmark patch from the selected channel image.

The first image may include a plurality of channel images; and theextracting of the landmark patch may include extracting the landmarkpatch from a channel image with a minimum wavelength difference betweenthe channel image and the second image among the plurality of channelimages.

The extracting of the landmark patch may include extracting the landmarkpatch from the first image for each of a plurality of landmarks of theobject, the extracting of the target patch may include extracting thetarget patch from the second image for each of the plurality oflandmarks, and the determining of the target point may includedetermining the target point corresponding to the landmark point foreach of the plurality of landmarks.

The extracting of the landmark patch may include determining an objectregion corresponding to the object in the first image; identifying thelandmark point of the object in the object region; and extracting thelandmark patch including the identified landmark point.

The method may further include matching the first image and the secondimage based on the landmark point of the first image and the targetpoint of the second image.

The method may further include preprocessing the landmark patch and thetarget patch using a Gaussian filter; and matching the preprocessedlandmark patch and the preprocessed target patch.

The extracting of the target patch may include determining the targetpatch in the second image based on a location of the landmark patch inthe first image and a distance between an image matching device and theobject.

The method may further include determining a point corresponding to aremaining landmark in the second image based on the target pointassociated with one of a plurality of landmarks of the object inresponse to the plurality of landmarks of the object being detected inthe first image.

The method may further include recognizing an object present in thesecond image based on the target point.

The method may further include verifying a liveness of the objectpresent in the second image based on the target point.

A non-transitory computer-readable storage medium may store instructionsthat, when executed by a processor, cause the processor to perform themethod.

In another general aspect, there is provided an image matching deviceincluding: one or more processors configured to: obtain a first image ofan object and a second image of the object; extract, from the firstimage, a landmark patch including a landmark point of the object,extract, from the second image, a target patch corresponding to thelandmark patch, and determine a target point in the second imagecorresponding to the landmark point based on a matching between thelandmark patch and the target patch.

The device may further include one or more image sensors configured toacquire the first image and the second image for the obtaining of thefirst image and the second image.

In another general aspect, there is provided an image matching methodincluding: extracting, from a first image of an object, a first patchincluding a first feature point of the object; extracting, from a secondimage of the object, a second patch based on the first patch;determining a second feature point in the second patch; and identifyingthe object or verifying an identity of the object based on the secondfeature point and the second image.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims. Additional aspects will beset forth in part in the description which follows and, in part, will beapparent from the description, or may be learned by practice of thepresented embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1 and 2 are flowcharts illustrating examples of an image matchingmethod.

FIG. 3 is a flowchart illustrating an example of an object recognitionand a liveness verification based on a determined target point.

FIG. 4 is a flowchart illustrating an example of a method of matching acolor image and an infrared (IR) image.

FIG. 5 illustrates an example of an image matching process and a featurepoint extraction.

FIGS. 6 and 7 illustrate examples of an image matching result.

FIG. 8 is a diagram illustrating an example of an image matching device.

FIGS. 9 and 10 illustrate examples of applying an image matching device.

Throughout the drawings and the detailed description, unless otherwisedescribed or provided, the same drawing reference numerals will beunderstood to refer to the same elements, features, and structures. Thedrawings may not be to scale, and the relative size, proportions, anddepiction of elements in the drawings may be exaggerated for clarity,illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, devices, and/orsystems described herein. However, various changes, modifications, andequivalents of the methods, devices, and/or systems described hereinwill be apparent after an understanding of the disclosure of thisapplication. For example, the sequences of operations described hereinare merely examples, however, are not limited to those set forth herein,but may be changed as will be apparent after an understanding of thedisclosure of this application, with the exception of operationsnecessarily occurring in a certain order. Also, descriptions of featuresthat are known may be omitted for increased clarity and conciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, devices, and/or systems described herein that will be apparentafter an understanding of the disclosure of this application.

Throughout the specification, when an element, such as a layer, region,or substrate, is described as being “on,” “connected to,” or “coupledto” another element, it may be directly “on,” “connected to,” or“coupled to” the other element, or there may be one or more otherelements intervening therebetween. In contrast, when an element isdescribed as being “directly on,” “directly connected to,” or “directlycoupled to” another element, there can be no other elements interveningtherebetween.

As used herein, the term “and/or” includes any one and any combinationof any two or more of the associated listed items.

Although terms such as “first,” “second,” and “third” may be used hereinto describe various members, components, regions, layers, or sections,these members, components, regions, layers, or sections are not to belimited by these terms. Rather, these terms are only used to distinguishone member, component, region, layer, or section from another member,component, region, layer, or section. Thus, a first member, component,region, layer, or section referred to in examples described herein mayalso be referred to as a second member, component, region, layer, orsection without departing from the teachings of the examples.

The terminology used herein is for describing various examples only, andis not to be used to limit the disclosure. The articles “a,” “an,” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. The terms “comprises,” “includes,”and “has” specify the presence of stated features, numbers, operations,members, elements, and/or combinations thereof, but do not preclude thepresence or addition of one or more other features, numbers, operations,members, elements, and/or combinations thereof.

The features of the examples described herein may be combined in variousways as will be apparent after an understanding of the disclosure ofthis application. Further, although the examples described herein have avariety of configurations, other configurations are possible as will beapparent after an understanding of the disclosure of this application.

Unless otherwise defined, all terms, including technical and scientificterms, used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure pertains and inview of the present disclosure. Terms, such as those defined in commonlyused dictionaries, are to be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and thepresent disclosure, and are not to be interpreted in an idealized oroverly formal sense unless expressly so defined herein.

Also, in the description of embodiments, detailed description ofwell-known related structures or functions will be omitted when it isdeemed that such description will cause ambiguous interpretation of thedisclosure.

FIGS. 1 and 2 are flowcharts illustrating examples of an image matchingmethod.

FIG. 1 is a flowchart illustrating an example of an image matchingmethod.

Referring to FIG. 1 , in operation 110, an image matching deviceextracts a landmark patch including a landmark point of an object from afirst image. The first image refers to an image captured by a firstimage sensor. For example, the first image may be a color image.However, the first image is not limited thereto. Also, the first imagemay be a black-and-white image and/or a depth image.

The color image includes a plurality of color channel images. A colorchannel image may represent an intensity at which light with awavelength band corresponding to a corresponding color is acquired bythe first image sensor, for example, a color sensor. For example, thecolor image includes a red channel image, a green channel image, and ablue channel image. The red channel image represents an intensity atwhich light with a wavelength band corresponding to a red color isreceived. The green channel image represents an intensity at which lightwith a wavelength band corresponding to a green color is received. Theblue channel image represents an intensity at which light with awavelength band corresponding to a blue color is received.

The depth image refers to an image captured by the first image sensor,for example, a depth sensor. Each pixel of the depth image has a valueindicating a distance between a corresponding pixel and a pointcorresponding to the pixel. The depth sensor may include a sensor basedon a time-of-flight (ToF) scheme, a sensor based on a structured lightscheme, and the like. However, the depth sensor is not limited thereto.

Herein, the term “landmark” may represent a desired part with respect tothe object. For example, if the object is a face of a human, thelandmark may be an eye, a nose, a mouth, an eye brow, lips, a pupil, andthe like. Also, the term “landmark point” may refer to a feature pointthat represents each feature of the object. For example, the landmarkpoint may be used to indicate both ends of one eye, a center of the eye,both ends of the mouth, a tip of the nose, and the like. However, thelandmark point is not limited thereto. The term “landmark patch” mayrefer to an image patch that includes the landmark point in the firstimage. The term “image patch” may refer to a subgroup of pixels of animage or a cropped or processed portion of the image.

In operation 120, the image matching device extracts a target patchcorresponding to the landmark patch from a second image. The secondimage refers to an image captured by a second image sensor. For example,the second image may be an infrared (IR) image. However, the secondimage is not limited thereto.

The target patch may be an image patch that includes a pointcorresponding to a location of the landmark patch. The image matchingdevice coarsely determines a location in the second image correspondingto the location of the landmark point or the location of the landmarkpatch in the first image. The image matching device extracts the targetpatch with a size greater than a size of the landmark patch. Anoperation of determining the target patch will be further describedbelow.

Herein, the image patch may refer to a set of pixels of a partial regionin an entire image. For example, the landmark patch may be a set ofpixels of a partial region in the first image. The target patch may be aset of pixels of a partial region in the second image.

An image type of the first image and an image type of the second imagemay be different. For example, the first image may be a color image, andthe second image may be an IR image. The first image may be an IR image,and the second image may be a color image. The first image may be adepth image, and the second image may be an IR image. The first imagemay be an IR image, and the second image may be a depth image. However,the image type of each of the first image and the second image is notlimited thereto. The image type of the first image and the image type ofthe second image may be identical. Both the first image and the secondimage may be color images. Both the first image and the second image maybe IR images. The image type of the first image and the image type ofthe second image may be classified, for example, based on a wavelengthof light that may be received by each of the first image sensor and thesecond image sensor configured to capture the respective correspondingimages. A first wavelength band image of light receivable by the firstimage sensor and a second wavelength band of light receivable by thesecond image sensor may be different. Although a description is madeherein using a color image and an IR image as an image type, they areprovided as examples only. For convenience of description, the followingdescription is made based on an example in which the color image is usedas the first image and the IR image is used as the second image.However, the first image and the second image are not limited thereto.

In operation 130, the image matching device determines a target pointcorresponding to the landmark point in the second image based on amatching between the landmark patch and the target patch. The imagematching device retrieves a partial region that matches the landmarkpatch from the target patch. For example, the image matching device maydivide the target patch into a plurality of partial regions and mayselect a partial region that is most similar to the landmark patch fromthe plurality of partial regions. The image matching device maydetermine a point of the partial region selected from the second image,for example, a center point, as the target point corresponding to thelandmark point in the first image.

The image matching device may adjust the target patch and the landmarkpatch at an identical resolution by performing scaling on at least oneof the target patch and the landmark patch, and may perform a patchmatching. For example, the image matching device may generate aplurality of partial regions by dividing the target patch into windowseach with a desired size. The image matching device may perform scalingon at least one of the partial regions and the landmark so that a size,a resolution, and a number of pixels of the partial region may becomeidentical to a size, a resolution, and a number of pixels of thelandmark patch, respectively.

Thus, the image matching device may efficiently determine a featurepoint with respect to the second image using a feature point identifiedin the first image without additionally extracting the feature pointfrom the second image.

FIG. 2 is a flowchart illustrating an image matching method.

Referring to FIG. 2 , in operation 210, an image matching deviceacquires a first image. For example, the image matching device mayacquire the first image by photographing or capturing an object using afirst image sensor.

In operation 220, the image matching device determines an object regionin the first image. In an example, the image matching device maydetermine the object region corresponding to the object in the firstimage based on an object model. The object model may be a model that isconfigured or trained to output a region corresponding to an object froman input image and, for example, may be a trained neural network.

The neural network represents a recognition model using a large numberof nodes that may be connected by edges, e.g., by weighted connections,and/or that may apply trained kernels, e.g., in implementedconvolutional operations. The neural network is implemented throughhardware or a combination of hardware and instructions, e.g., throughinstructions stored in a non-transitory memory of the image matchingdevice, which when executed by one or more processors of the imagematching device, cause the one or more processors to implement therecognition model. The recognition model may be trained in the memory ofthe image matching device in various structures. The various datastructures may include storing the resulting trained parameters, e.g.,including the trained connection weights and/or kernels, in vector,matrix, volume, or other single or multi-dimensional data structure.Also, though the recognition model is discussed using the example neuralnetwork structure, alternate machine learning structures may also beavailable in other examples.

In operation 230, the image matching device extracts a landmark patchbased on the object region. In an example, the image matching device mayidentify a landmark point of the object in the object region. Forexample, the image matching device may determine the landmark point inthe object region based on a landmark model. The landmark model may be amodel that outputs the landmark point from the object region and, forexample, may also be a trained neural network, as a non-limitingexample, trained for the landmark patch extraction objective. The imagematching device may extract the landmark patch that includes theidentified landmark point. For example, the image matching device mayextract, as the landmark patch, an image patch using the landmark pointas a center point.

In operation 240, the image matching device acquires a second image. Forexample, the image matching device acquires the second image byphotographing the object using a second image sensor. A point in time atwhich the first image is captured and a point in time at which thesecond image is captured may be different. For example, the imagematching device may acquire the first image and the second image atdifferent timings, respectively. Also, the image matching device maysimultaneously acquire the first image and the second image. Also, theimage matching device may adjust scales between two images based ontheir respective fields of view (FOVs) and a relative locationaldifference between the first image sensor and the second image sensor,as non-limiting examples.

In operation 250, the image matching device determines a target patch inthe second image. The image matching device may determine the targetpatch in the second image based on a location of the landmark patch inthe first image. Also, the image matching device may determine thetarget patch in the second image based on the location of the landmarkpatch in the first image and a distance between the object and the imagematching device. For example, the image matching device may determine,as the target patch, an image patch in the second image that is mappedwith respect to the location of the landmark patch in the first image.The image matching device may estimate a region in which a feature pointcorresponding to the landmark is expected to be present in the secondimage, based on the location of the landmark patch in the first image.

Also, the image matching device extracts the target patch in response tothe landmark point being extracted from a desired region (e.g., adetermined region) of the first image. The image matching devicedetermines the desired region based on a disposition between the firstimage sensor capturing the first image and the second image sensorcapturing the second image, a FOV of the first image sensor, and a FOVthe second image sensor. For example, the image matching device maydetermine a region corresponding to the FOV of the second image sensorin the first image as the desired image based on a separate distancebetween the first image sensor and the second image sensor and adistance between the object and the image matching device. The distancebetween the object and the image matching device may be a predetermined(or, alternatively, desired) distance, for example, 15 cm to 30 cm. Forexample, the image matching device may measure a distance from theobject using a depth sensor or estimate the distance from the objectusing a disparity between a left image and a right image acquired by astereo camera.

The image matching device terminates an image matching operation inresponse to a failure in identifying the landmark point. Also, the imagematching device terminates the image matching operation in response tothe landmark point identified in the first image being absent in thedesired region.

In operation 260, the image matching device matches the landmark patchand the target patch. The image matching device retrieves a partialregion that matches the landmark patch from the target patch. An exampleof retrieving the partial region that matches the landmark patch will befurther described with reference to FIG. 5 .

In operation 270, the image matching device determines a target pointbased on an image matching result. For example, the image matchingdevice determines a center point of the retrieved partial region as thetarget point. As described above, the first image and the second imagemay be acquired at different timings, respectively. Thus, a form, alocation, and a pose of the object present in the first image may bedifferent from those of the object present in the second image. In anexample, the image matching device may accurately match images capturedat different points in times by matching the first image and the secondimage based on the landmark.

FIG. 3 is a flowchart illustrating an example of an object recognitionand a liveness verification based on a determined target point.

Referring to FIG. 3 , in operation 340, the image matching devicerecognizes the object or verifies a liveness of the object based on atarget point. For example, the image matching device may recognize theobject that is present in the second image based on a target point ofthe second image. As another example, the image matching device mayverify an effectiveness of the liveness of the object that is present inthe second image based on the target point of the second image.

Herein, the term “recognition” may include a verification and anidentification. As non-limiting examples, the verification may representan operation of determining whether input data is true or false, and therecognition may represent an operation of determining a label indicatedby the input data among a plurality of labels.

The liveness represents whether the object is an actual living body andnot a spoof attempt or fake/forged object. For example, the imagematching device determines the liveness of the object based on aliveness parameter determined for each frame of at least one of thefirst image and the second image. The liveness parameter is used todetermine whether each frame is based on an image that is effectivelycaptured from an actual user. For example, the liveness parameterrepresents “true” if an image corresponding to a frame is captured fromthe actual user. The liveness parameter represents “false” if the imageis forged. The image matching device may calculate the livenessparameter based on at least one of a feature point of the first imageand a feature point of the second image.

FIG. 4 is a flowchart illustrating an example of a method of matching acolor image and an IR image.

In operation 410, an image matching device acquires a color image as afirst image. In operation 440, the image matching device acquires an IRimage as a second image. The image matching device may capture the colorimage and the IR image from an identical object. That is, the identicalobject may be included in both the color image and the IR image.

In operation 420, the image matching device detects a face and a facefeature point. For example, the image matching device may detect a faceregion based on the color image. The image matching device may detectthe face feature point in the face region.

In operation 430, the image matching device extracts a patch around theface feature point. For example, the image matching device may extract alandmark patch including the face feature point detected in operation420.

In operation 450, the image matching device extracts a patch around alocation corresponding to the face feature point. For example, the imagematching device may extract a target patch from the IR imagecorresponding to the face feature point detected in operation 420.

In operation 431, the image matching device preprocesses the colorimage. In operation 451, the image matching device preprocesses the IRimage. The image matching device may apply a Gaussian filter to thecolor image and the IR image. The Gaussian filter may output a magnitudeof gradient with respect to pixels of an image. A pixel value of eachpixel of the image to which the Gaussian filter is applied may representa Gaussian magnitude. The color image to which the Gaussian filter isapplied may be blurred. In the IR image to which the Gaussian filter isapplied, an edge of the IR image may be emphasized. For example, theimage matching device may preprocess the landmark patch and the targetpatch using the Gaussian filter. The image matching device may perform amorphology operation with respect to the IR image before the Gaussianfilter is applied. The morphology operation may reduce an effect byspeckles or spotlights occurring due to reflection by eyeglasses, whichmay be present in the IR image.

Although preprocessing operations in operations 431 and 451 of FIG. 4are illustrated to be followed by operations 430 and 450, respectively,it is provided as an example only. The image matching device may performthe preprocessing operations before operations 430 and 450 and may alsoperform the preprocessing operations before detecting the face inoperation 420.

In operation 460, the image matching device matches patches between twoimages, that is, the color image and the IR image. For example, theimage matching device may determine a partial region of the target patchin the IR image that matches the landmark patch in the color image. Theimage matching device may match the preprocessed landmark patch and thepreprocessed target patch.

In operation 470, the image matching device acquires the face featurepoint for the IR image. In an example, the image matching device maydetermine the face feature point with respect to the IR image based onthe aforementioned matching between the landmark patch and the targetpatch. The image matching device may determine a point included in thedetermined partial region. For example, a center point of the partialregion may be determined as a target point. The target point may be theface feature.

Thus, even though the IR image may be vulnerable to reflection by anaccessory having a characteristic of total reflection, for example,eyeglasses (and therefore it may be difficult to accurately determinethe feature point directly from the IR image), the image matching devicemay accurately determine the feature point with respect to the IR imageby mapping the feature point extracted based on the color image withrespect to the IR image.

FIG. 5 illustrates an example of an image matching process and a featurepoint extraction.

Although FIG. 5 illustrates an example in which a color image 510 isused as a first image and an IR image 520 is used as a second image, itis provided as an example only. For example, the first image may be anIR image, and the second image may be a color image. Both the firstimage and the second image may be IR images. Both the first image andthe second image may be color images. Further, the first image may be adepth image, and the second image may be a color image. For example, thefirst image may be one of a color image, an IR image, and a depth image,and the second image may be one of a color image, an IR image, and adepth image.

An image matching device acquires the color image 510 as the firstimage. Referring to FIG. 5 , a first image sensor of the image matchingdevice generates the color image 510 by photographing a face of a humanas an object.

The image matching device identifies a landmark point of the object withrespect to the color image 510. For example, the image matching deviceextracts a feature point of the face of the human from the color image510 based on an object model. In an example, the image matching devicemay determine whether the landmark point is identified in apredetermined (or, alternatively, desired) region 511. The landmarkpoint is represented as a dot in FIG. 5 . The region 511 in the colorimage 510 is a region corresponding to a FOV of the IR image 520. Thus,in response to the landmark point extracted from the color image 510being absent in the region 511, a landmark of the object, for example, anose of the face, may be absent in the IR image 520. In response to thelandmark of the object being absent in the IR image 520, alandmark-based matching may not be performed. Thus, the image matchingdevice terminates a matching operation.

The image matching device acquires the IR image 520 as the second image.A second image sensor of the image matching device generates the IRimage 520 by photographing the face of the human as the object. Thefirst image sensor may be a camera sensor, and the second image sensormay be an IR sensor. A FOV of the IR sensor may be narrower or widerthan that of the camera sensor. The camera sensor acquires the colorimage in a visible band, and the IR sensor acquires the IR image in anIR band. Referring to FIG. 5 , a ratio of the face in the IR image 520is relatively great compared to the color image 510.

The first image that is acquired by the image matching device includes aplurality of color channel images. For example, the plurality of colorchannel images may include a red channel image, a green channel image,and a blue channel image. Also, the first image may include a brightnesschannel image, for example, a Y channel image, and a chrominance channelimages, for example, a U channel image and a V channel image.

The image matching device extracts a landmark patch 531 from a channelimage 530 of the plurality of color channel images with a minimumwavelength difference with respect to the second image in response tothe first image including the plurality of the color channel images. Forexample, if the first image includes the red channel image, the greenchannel image, and the blue channel image, and the second image is theIR image 520, the color channel image with the minimum wavelengthdifference is the red channel image. Thus, the image matching deviceselects the red channel image from the first image as the channel image530. The image matching device extracts the landmark patch 531 from thechannel image 530. The channel image 530 may be a preprocessed colorimage, as shown in FIG. 5 . Preprocessing of the color image 510 may beperformed using, for example, a Gaussian filter.

The image matching device extracts a target patch 541 of a locationcorresponding to the landmark patch 531 of the color image 510 from theIR image 520. For example, the image matching device extracts the targetpatch 541 from a preprocessed IR image 540.

The image matching device retrieves a partial region that matches thelandmark patch 531 from the target patch 541. The image matching devicedetermines a center point of the retrieved partial region as a targetpoint. For example, the image matching device may calculate a similaritylevel (for example, a correlation level) between the landmark patch 531and each of a plurality of partial regions of the target patch 541. Theimage matching device may determine a partial region with a highestcalculated similarity level (for example, correlation level) among theplurality of partial regions as a partial region that matches thelandmark patch 531.

The image matching device calculates a similarity level between pixelsincluded in each of the plurality of partial regions of the target patch541 and pixels included in the landmark patch 531. The image matchingdevice calculates the correlation level, for example, across-correlation score, as the similarity level. For example, the imagematching device may calculate a normalized cross-correlation value asthe cross-correlation score by performing a normalized cross-correlationoperation with respect to two patches. Referring to FIG. 5 , the imagematching device calculates a patch 550 in which a partial region of thetarget patch 541 and the landmark patch 531 are cross-correlated. Astatistical value, for example, an average value and a median value,with respect to pixel values of pixels included in the cross-correlatedpatch 550 may be determined as the correlation level.

To improve a calculation rate, the image matching device samples aportion of the target patch 541 and calculates a similarity levelbetween the sampled portion and the landmark patch 531. For example, thesimilarity level may be a value corresponding to the correlation level.

The image matching device determines a target point 561 with respect tothe IR image 520 based on a matching between the landmark patch 531 andthe target patch 541. The target point 561 may represent a feature pointof the object, for example, the face of the human, present in the IRimage 520. The image matching device recognizes the object or verifies aliveness of the object based on a second IR image 560 in which thetarget point 561 is determined. The second IR image 560 may be the sameimage as the IR image 520 (but the second IR image 560 is not limitedthereto, and the second IR image may be different from the IR image520).

In an example, the image matching device may match a first image and asecond image based on a landmark point of the first image and a targetpoint of the second image. For example, the image matching device maymatch the landmark point and the target point by transforming at leastone of the first image and the second image based on the landmark pointof the first image and the target point of the second image. The imagematching device may match the landmark point and the target point byapplying, for example, transform, shift, and rotation, to at least oneof the first image and the second image. With respect to remainingpixels excluding the feature points, for example, the landmark point andthe target point, in the first image and the second image, matching maybe performed by applying transformation based on the landmark point andthe target point.

FIGS. 6 and 7 illustrate examples of an image matching result.

FIG. 6 illustrates an example in which a portion of a landmark of anobject is present in a predetermined (or, alternatively, desired) region611 in a color image 610 captured by an image matching device. Since aFOV of an IR image 620 is less than that of the color image 610, only aportion of the object may be present in the IR image 620. In an example,in response to only the portion of the landmark being identified in thedesired region 611 of the color image 610, the image matching device maydetermine a target point corresponding to the identified landmark in theIR image 620. Thus, the image matching device may accurately determine afeature point corresponding to the landmark based on a landmark point ofthe color image 610 although only the portion of the landmark is presentin the FOV of the IR image 620.

FIG. 7 illustrates an example in which a plurality of landmarks ispresent in a predetermined (alternatively, desired) region 711 in afirst image 710 captured by an image matching device. The image matchingdevice extracts a landmark patch for each of a plurality of landmarks ofan object from the first image 710. The image matching device extracts atarget patch for each of the plurality of landmarks from a second image720. The image matching device determines a target point correspondingto the landmark point for each of the plurality of landmarks.

Without being limited thereto, the image matching device determines apoint corresponding to a remaining landmark in the second image 720based on the target point associated with one of the plurality oflandmarks in response to the plurality of landmarks being detected withrespect to the object from the first image 710. Thus, although targetpoints with respect to entire landmarks of the second image 720 are notdetermined, the image matching device may estimate a feature pointcorresponding to the remaining landmark based on target pointsdetermined with respect to a portion of landmarks.

In an example, the image matching device may calculate a distancebetween a landmark point with respect to a landmark in a first image anda target point corresponding thereto in a second image. The imagematching device may estimate a location of a feature point in the secondimage by reflecting the calculated distance to a feature point in thefirst image corresponding to a remaining landmark.

In an example, the image matching device may determine a correctedlocation of a feature point corresponding to a remaining landmark in acolor image as a location of a feature point in an IR image, based on adistance calculated with respect to a landmark in the color image. Inresponse to pixels in a desired region, for example, a regioncorresponding to a FOV of the IR image, in the color image being simplymapped to pixels in the IR image based on a one-to-one correspondence,for example, pixel-by-pixel, a disparity may occur between an objectpresent in the color image and the object present in the IR image basedon a relative location between the object and the image matching device.Therefore, a pixel determined as a feature point in the IR image mayhave an error. The image matching device may accurately determine afeature point in the IR image by correcting the disparity.

In an example, the image matching device may determine a target point ofa second image based on a matching between a landmark patch and a targetpatch. Therefore, the image matching device may accurately determine afeature point of the second image compared to a case of simplytransforming a pixel location of the first image to a pixel location ofthe second image.

FIG. 8 is a diagram illustrating an example of an image matching device.In an example, the image matching device may be the image matchingdevice discussed above in any or any combination of FIGS. 1 through 7 ,wherein examples are not limited thereto.

Referring to FIG. 8 , an image matching device 800 includes an imageacquirer 810 and a processor 820. Also, the image matching device 800further includes a memory 830. The processor 820 and memory 830 are alsorespectively representative of one or more processors 820 and one ormore memories 830.

The image acquirer 810 acquires a first image and a second image. Theimage acquirer 810 includes a first image sensor and a second imagesensor. The first image sensor acquires the first image, and the secondimage sensor acquires the second image. The first image sensor and thesecond image sensor may have different sensor characteristics, forexample, may have different FOVs. Wavelength bands, for example, avisible band and an IR band, in which the first image and the secondimage are captured respectively, may be different. Also, points in timesat which the first image and the second image are captured,respectively, may be different. However, they are provided as examplesonly. Types of the first image and the second image may be identical.Alternatively, the points in times at which the first image and thesecond image are captured, respectively, may be identical.

The processor 820 extracts a landmark patch including a landmark pointof an object from the first image and extracts a target patchcorresponding to the landmark patch from the second image. The processor820 determines a target point corresponding to the landmark point in thesecond image based on a matching between the landmark patch and thetarget patch. However, an operation of the processor 820 is not limitedthereto, and the processor 820 may perform the operations describedabove with reference to FIGS. 1 through 7 . Also, without being limitedto the description made above with reference FIG. 1 thorough 7, theoperations may be performed in various orders in various examples.

The memory 830 temporally and semi-permanently stores data used toperform the image matching method. For example, the memory 830 storesimages generated during an image matching process. Also, the memory 830stores models, for example, an object model and the like, that are usedfor recognition and parameters.

In an example, the image matching device 800 does not require a facedetector and a face feature point detector for each image type. If atype of an image is different, for example, if the image is a colorimage or an IR image, the image matching device 800 detects a featurepoint with respect to the different type of the image using a featurepoint detector developed with respect to a single image.

Also, the image matching device 800 performs a matching with respect toa partial region that is expected as a feature point and thus may savecomputing power. Also, the image matching device 800 may reduce anamount of time used for matching.

FIGS. 9 and 10 illustrate examples of applying an image matching device.In an example, the image matching device may be the image matchingdevice discussed above in any or any combination of FIGS. 1 through 8 ,wherein examples are not limited thereto.

Referring to FIG. 9 , the image matching device is representative of, orapplied to, a device 900, for example, a smartphone, including amulti-image sensor. For example, the image matching device may berepresentative of, or applied to, an authentication system using amulti-modal input image. The multi-image sensor may detect either one orboth of color images and IR images.

Referring to FIG. 10 , the image matching device may be representativeof, or applied to, a security management system 1000, for example, aclosed-circuit television (CCTV). For example, the image matching devicemay accurately determine a feature point of an object, complexly using afirst image and a second image although an amount of light is limited,such as night.

The image matching device 800, the image acquirer 810, the processor820, the memory 830, the device 900, and other components describedherein with respect to FIGS. 1 through 10 are implemented by hardwarecomponents. Examples of hardware components that may be used to performthe operations described in this application where appropriate includecontrollers, sensors, generators, drivers, memories, comparators,arithmetic logic units, adders, subtractors, multipliers, dividers,integrators, and any other electronic components configured to performthe operations described in this application. In other examples, one ormore of the hardware components that perform the operations described inthis application are implemented by computing hardware, for example, byone or more processors or computers. A processor or computer may beimplemented by one or more processing elements, such as an array oflogic gates, a controller and an arithmetic logic unit, a digital signalprocessor, a microcomputer, a programmable logic controller, afield-programmable gate array, a programmable logic array, amicroprocessor, or any other device or combination of devices that isconfigured to respond to and execute instructions in a defined manner toachieve a desired result. In one example, a processor or computerincludes, or is connected to, one or more memories storing instructionsor software that are executed by the processor or computer. Hardwarecomponents implemented by a processor or computer may executeinstructions or software, such as an operating system (OS) and one ormore software applications that run on the OS, to perform the operationsdescribed in this application. The hardware components may also access,manipulate, process, create, and store data in response to execution ofthe instructions or software. For simplicity, the singular term“processor” or “computer” may be used in the description of the examplesdescribed in this application, but in other examples multiple processorsor computers may be used, or a processor or computer may includemultiple processing elements, or multiple types of processing elements,or both. For example, a single hardware component or two or morehardware components may be implemented by a single processor, or two ormore processors, or a processor and a controller. One or more hardwarecomponents may be implemented by one or more processors, or a processorand a controller, and one or more other hardware components may beimplemented by one or more other processors, or another processor andanother controller. One or more processors, or a processor and acontroller, may implement a single hardware component, or two or morehardware components. A hardware component may have any one or more ofdifferent processing configurations, examples of which include a singleprocessor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods that perform the operations described in this applicationare performed by computing hardware, for example, by one or moreprocessors or computers, implemented as described above executinginstructions or software to perform the operations described in thisapplication that are performed by the methods. For example, a singleoperation or two or more operations may be performed by a singleprocessor, or two or more processors, or a processor and a controller.One or more operations may be performed by one or more processors, or aprocessor and a controller, and one or more other operations may beperformed by one or more other processors, or another processor andanother controller. One or more processors, or a processor and acontroller, may perform a single operation, or two or more operations.

Instructions or software to control computing hardware, for example, oneor more processors or computers, to implement the hardware componentsand perform the methods as described above may be written as computerprograms, code segments, instructions or any combination thereof, forindividually or collectively instructing or configuring the one or moreprocessors or computers to operate as a machine or special-purposecomputer to perform the operations that are performed by the hardwarecomponents and the methods as described above. In one example, theinstructions or software include machine code that is directly executedby the one or more processors or computers, such as machine codeproduced by a compiler. In another example, the instructions or softwareincludes higher-level code that is executed by the one or moreprocessors or computer using an interpreter. The instructions orsoftware may be written using any programming language based on theblock diagrams and the flow charts illustrated in the drawings and thecorresponding descriptions used herein, which disclose algorithms forperforming the operations that are performed by the hardware componentsand the methods as described above.

The instructions or software to control computing hardware, for example,one or more processors or computers, to implement the hardwarecomponents and perform the methods as described above, and anyassociated data, data files, and data structures, may be recorded,stored, or fixed in or on one or more non-transitory computer-readablestorage media. Examples of a non-transitory computer-readable storagemedium include read-only memory (ROM), random-access memory (RAM), flashmemory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs,DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetictapes, floppy disks, magneto-optical data storage devices, optical datastorage devices, hard disks, solid-state disks, and any other devicethat is configured to store the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and provide the instructions or software and any associated data,data files, and data structures to one or more processors or computersso that the one or more processors or computers can execute theinstructions. In one example, the instructions or software and anyassociated data, data files, and data structures are distributed overnetwork-coupled computer systems so that the instructions and softwareand any associated data, data files, and data structures are stored,accessed, and executed in a distributed fashion by the one or moreprocessors or computers.

While this disclosure includes specific examples, it will be apparentafter an understanding of the disclosure of this application thatvarious changes in form and details may be made in these exampleswithout departing from the spirit and scope of the claims and theirequivalents. The examples described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each example are to be considered as beingapplicable to similar features or aspects in other examples. Suitableresults may be achieved if the described techniques are performed in adifferent order, and/or if components in a described system,architecture, device, or circuit are combined in a different manner,and/or replaced or supplemented by other components or theirequivalents. Therefore, the scope of the disclosure is defined not bythe detailed description, but by the claims and their equivalents, andall variations within the scope of the claims and their equivalents areto be construed as being included in the disclosure.

What is claimed is:
 1. A processor-implemented image matching method,comprising: photographing, by a first image sensor, an object to acquirea first image; photographing, by a second image sensor, the object toacquire a second image; determining a landmark point of the first imagebased on a neural network model stored in a memory; extracting, from thefirst image of the object, a landmark patch including the landmark pointof the object; extracting, from the second image of the object, a targetpatch corresponding to the landmark patch, wherein a size of the targetpatch is greater than a size of the landmark patch; and determining atarget point in the second image corresponding to the landmark pointbased on a matching between the landmark patch extracted from the firstimage and the target patch extracted from the second image.
 2. The imagematching method of claim 1, wherein the photographing by the first imagesensor comprises acquiring, using a color image sensor, a color image asthe first image; and the photographing by the second image sensorcomprises acquiring, using an infrared (IR) image sensor, an IR image asthe second image.
 3. The image matching method of claim 1, furthercomprising: determining, based on the target point, whether the objectis an anatomical structure of the user.
 4. The image matching method ofclaim 1, further comprising: allowing access, through the user interfaceof a device, to one or more features of the device in response to theobject being determined to be either one or both of a live anatomicalstructure and a recognized user.
 5. The image matching method of claim1, wherein the extracting of the target patch comprises determining thetarget patch in the second image based on a location of the landmarkpatch in the first image.
 6. The image matching method of claim 1,wherein the extracting of the target patch comprises extracting thetarget patch in response to the landmark point being detected in adetermined region of the first image.
 7. The image matching method ofclaim 6, wherein the extracting of the target patch comprisesdetermining the determined region based on a difference between a fieldof view (FOV) of a first image sensor used to capture the first imageand an FOV of a second image sensor used to capture the second image. 8.The image matching method of claim 1, wherein the determining of thetarget point comprises: retrieving, from the target patch, a partialregion that matches the landmark patch; and determining a center pointof the retrieved partial region as the target point.
 9. The imagematching method of claim 8, wherein the retrieving of the partial regioncomprises: calculating a similarity level between the landmark patch andeach of a plurality of partial regions of the target patch; anddetermining a partial region with a highest calculated similarity levelamong the plurality of partial regions as the partial region thatmatches the landmark patch.
 10. The image matching method of claim 9,wherein the calculating of the similarity level comprises calculating acorrelation level between values of pixels included in each of theplurality of partial regions of the target patch and pixels included inthe landmark patch as the similarity level.
 11. The image matchingmethod of claim 1, wherein: the first image is a color image and thesecond image is an infrared (IR) image; and the extracting of thelandmark patch comprises: selecting a channel image from the firstimage; and extracting the landmark patch from the selected channelimage.
 12. The image matching method of claim 1, wherein: the firstimage includes a plurality of channel images; and the extracting of thelandmark patch comprises extracting the landmark patch from a channelimage with a minimum wavelength difference between the channel image andthe second image among the plurality of channel images.
 13. The imagematching method of claim 1, wherein: the extracting of the landmarkpatch comprises extracting the landmark patch from the first image foreach of a plurality of landmarks of the object, the extracting of thetarget patch comprises extracting the target patch from the second imagefor each of the plurality of landmarks, and the determining of thetarget point comprises determining the target point corresponding to thelandmark point for each of the plurality of landmarks.
 14. The imagematching method of claim 1, wherein the determining of the landmarkpoint comprises: determining an object region corresponding to theobject in the first image; identifying the landmark point of the objectin the object region.
 15. The image matching method of claim 1, furthercomprising: matching the first image and the second image based on thelandmark point of the first image and the target point of the secondimage.
 16. The image matching method of claim 1, wherein the extractingof the target patch comprises determining the target patch in the secondimage based on a location of the landmark patch in the first image and adistance between an image matching device and the object.
 17. The imagematching method of claim 1, further comprising: determining a pointcorresponding to a remaining landmark in the second image based on thetarget point associated with one of a plurality of landmarks of theobject in response to the plurality of landmarks of the object beingdetected in the first image.
 18. The image matching method of claim 1,further comprising: recognizing the object, in the second image, basedon the target point; and authenticating a user based on the recognizedobject, wherein the landmark point comprises both ends of an eye of theuser.
 19. The image matching method of claim 1, further comprising:verifying a liveness of the object, in the second image, based on thetarget point.
 20. An image matching device, comprising: a first imagesensor configured to capture an object for acquiring a first image; asecond image sensor configured to capture the object for acquiring asecond image; a memory configured to store a neural network model; andone or more processors configured to: determine a landmark point of thefirst image based on the neural network model stored in the memory,obtain the first image of the object and the second image of the object;extract, from the first image, a landmark patch including the landmarkpoint of the object, extract, from the second image, a target patchcorresponding to the landmark patch, and determine a target point in thesecond image corresponding to the landmark point based on a matchingbetween the landmark patch extracted from the first image and the targetpatch extracted from the second image, and wherein a size of the targetpatch is greater than a size of the landmark patch.